A comprehensive method for solving finite-state semi-Markov processes

Inference strategies for solving semi-Markov decision processes

Semi-Markov decision processes are used to formulate many control problems and also play a key role in hierarchical reinforcement learning. In this chapter we show how to translate the decision making problem into a form that can instead be solved by inference and learning techniques. In particular, we will establish a formal connection between planning in semiMarkov decision processes and infe...

The KTH Visit in Semi-Markov Processes

A nonlinear fixed-lag smoother for finite-state Markov processes

The fixed-lag smoothing of random telegraph type signals is studied. The smoothers are derived by first obtaining fixed-point smoothing equations and then using a time discretization. Simulation results are described that verify the qualitative carry-over of known results for the l inear-Gaussian problem: the greater the lag, the greater the improvement; beyond a certain lag, no further improve...

A finite difference method for piecewise deterministic Markov processes

An extension of non-deterministic processes driven by the random telegraph signal is introduced in the framework of piecewise deterministic Markov processes [9], including a broader category of random systems. The corresponding Liouville-Master Equation is established and the upwind method is applied to numerical calculation of the distribution function. The convergence of the numerical solutio...

Solving Generalized Semi-Markov Processes using Continuous Phase-Type Distributions

We introduce the generalized semi-Markov decision process (GSMDP) as an extension of continuous-time MDPs and semi-Markov decision processes (SMDPs) for modeling stochastic decision processes with asynchronous events and actions. Using phase-type distributions and uniformization, we show how an arbitrary GSMDP can be approximated by a discrete-time MDP, which can then be solved using existing M...